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SYMBOLIC DYNAMICS, AUTOMORPHIC FUNCTIONS, AND 
SELBERG ZETA FUNCTIONS WITH UNITARY 
REPRESENTATIONS 

ANKE D. POHL 


Abstract. Using Hecke triangle surfaces of finite and infinite area as exam¬ 
ples, we present techniques for thermodynamic formalism approaches to Sel- 
berg zeta functions with unitary finite-dimensional representations (V, x) for 
hyperbolic surfaces (orbifolds) r\H as well as transfer operator techniques to 
develop period-like functions for (E, x)-automorphic cusp forms. This leads to 
several natural conjectures. We further show how to extend these results to 
the billiard flow on the underlying triangle surfaces, and study the convergence 
of transfer operators along sequences of Hecke triangle groups. 


1. Introduction 

The intimate relation between the geometric and spectral properties of Riemannian 
locally symmetric spaces is of utmost interest in various areas, including dynamical 
systems, spectral theory, harmonic analysis, representation theory, number theory 
and quantum chaos, and contributes to their cross-fertilization. It is one reason for 
the increasing number of competing or complementary approaches via dynamical 
systems and ergodic theory on the one hand, and, e.g., harmonic analysis and 
analytic number theory on the other hand (see, for example, [9, 10] or [17, 35, 37]). 
Up to date, the full extent of this relation, its consequences and the properties 
of several entities on the geometric and the spectral side remain an active area of 
research. 

Over the last three decades, mostly for hyperbolic surfaces, an approach comple¬ 
mentary to the classical analytic and number-theoretic methods emerged within 
the framework of the thermodynamic formalism of statistical mechanics: transfer 
operator techniques. These techniques focus on the dynamics of the geodesic flows 
rather than on the geometry of the surfaces, which allowed to establish results, 
a few mentioned below, hitherto unattained by any other method, or to provide 
alternative proofs to known results. 

One such class of examples are classical dynamical approaches to Laplace eigen¬ 
functions, most notably Maass cusp forms, as well as representations of Selberg 
zeta functions as Fredholm determinants of transfer operator families, which allow 
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US to show their meromorphic continuations. The modular surface 

PSL2(Z)\H = PSL2(Z)\ PSL2(R)/ PS0(2) 

had been the first instance for which such transfer operator approaches could be 
established. For this, Mayer [18, 19] considered the Gauss map 

: [0,1] \ Q [0,1] \ Q, x - mod 1, 

X 

which is well-known to derive from a symbolic dynamics for the geodesic flow on 
the modular surface [1, 34]. He investigated its associated transfer operator Ck,s 
with parameter s € C, hence the operator 

CK,sf{x) = + , 

and found a Banach space B on which, for Res > 1/2, the operator Ck,s acts, is 
nuclear of order 0, and the map s i—^ Ck,s has a meromorphic extension to all of 
C (which we also denote by Ck,s)- The Selberg zeta function Z for the modular 
surface is then given as the product of the Fredholm determinants 

(1) Z'(s) =det(l-£i^,s)det(H~£ic,s)■ 

This provides an alternative proof of the meromorphic extension of the Selberg 
zeta function. Even more, it shows that the zeros of Z are determined by the 
eigenfunctions with eigenvalue ±1 of Ck,s in B. The natural question whether 
these eigenfunctions, for Res = 1/2, actually characterize the (even/odd) Maass 
cusp forms for PSL 2 (Z) could be answered affirmately. Both Lewis-Zagier [16] and 
Chang-Mayer [5] showed that these eigenfunctions are bijective to highly regular 
solutions of the functional equation 

(2) f{x) = f{x + l) + {x + l) a; G R>o 

and satisfy in addition a certain symmetry. Without using any Selberg theory, 
Lewis-Zagier [16] continued to show that these solutions are in bijection with the 
even/odd Maass cusp forms, which justifies to call them period functions (in anal¬ 
ogy to the period polynomials in Eichler-Shimura theory). In total, this transfer 
operator/thermodynamic formalism approach complements Selberg theory in the 
following sense: it provides a characterization of Maass cusp forms as eigenfunc¬ 
tions of transfer operators which are constructed using only the geodesic flow on 
the modular surface. Thus it gives a classical dynamical characterization of the 
Maass cusp forms themselves, not only of their spectral parameters. Moreover, by 
taking Fredholm determinants, it recovers the interpretation of (some of) the zeros 
of the Selberg zeta function. 

Such kinds of results are quite sensitive to the choice of the discretization and 
symbolic dynamics for the geodesic flow on the considered hyperbolic surface r\IHI. 
For subgroups F of PSL 2 (Z) of finite index they could be shown by “twisting” the 
PSL 2 (Z)-setup with the unitary representation of PSL 2 (Z) induced from the trivial 
one-dimensional representation of F [6, 7, 8]. In this way, the symbolic dynamics, 
the transfer operators and the functional equation get pushed from PSL 2 (Z) to F, 
and the representation essentially serves as a bookkeeping device for the cosets of 
F\PSL 2 (Z). An alternative symbolic dynamics for PSL 2 (Z) was used in [4]. All 
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other used discretizations and symbolic dynamics only led (yet) to a representation 
of the Selberg zeta function as a Fredholm determinant [32, 22, 11, 20]. 

In [26, 30] (see also [13]) we constructed discretizations for the geodesic flow on 
hyperbolic spaces r\IHI for Fuchsian groups F with at least one cusp and satisfying 
an additional (weak) geometric condition. These discretizations/symbolic dynam¬ 
ics are tailor-made for the requirements of transfer operator approaches to Maass 
cusp forms and Selberg zeta functions. They allowed us to develop the following re¬ 
definition of the structure of these approaches: We first construct a certain discrete 
dynamical system on the geodesic boundary of H and a certain symbolic dynamics 
on a finite alphabet (we call these systems “slow”). Hence the transfer operators 
associated to these systems have only finitely many terms, and their eigenfunctions 
are therefore obviously characterized by finite-term functional equations. If F is a 
lattice, then the highly regular eigenfunctions with eigenvalue 1 (period functions) 
are in bijection with the Maass cusp forms for F [21, 29, 27]. 

Thus, as in the seminal approach for PSL 2 (Z), we provide classical dynamical 
characterizations of the Maass cusp forms. But in contrast we do not need to make 
a detour to the Selberg zeta function and then hope for a finite-term functional 
equation to pop out of an infinite-term transfer operator. Our transfer operators 
provide us in an immediate and natural way with the necessary functional equations. 
For PSL 2 (Z) our transfer operators are 

= f(x -b 1) -b (x -b ■ 

They obviously reproduce the Lewis-Zagier functional equation (2). 

These slow systems are not uniformly expanding for which reason the finite-term 
transfer operators are not nuclear and cannot represent the Selberg zeta function 
as a Fredholm determinant. To overcome this obstacle, we apply an accelera¬ 
tion/induction procedure to provide a uniformly expanding discrete dynamical sys¬ 
tem and symbolic dynamics, which necessarily uses an infinite alphabet. We call 
these systems “fast”. The associated infinite-term transfer operators then represent 
the Selberg zeta function. Also here we recover Mayer’s transfer operator. In [21] 
this induction procedure is performed for cofinite Hecke triangle groups, in this 
article for Fo(2) respectively the Theta group, and in [31] for non-cofinite Hecke 
triangle groups. In a forthcoming article it will be shown for all admissible F. 

In [28] we used the idea of these parallel, but closely related, fast/slow systems for 
the billiard flow on the triangle surface underlying the Hecke triangle surfaces. We 
showed how specific weights of these systems allow us to accommodate Dirichlet 
resp. Neumann boundary value conditions, and in turn to geometrically separate 
odd and even Maass cusp forms. These results illuminate the factorization of the 
Selberg zeta function in (1) and explain the additional symmetries needed for the 
solutions of the functional equation (2). 

Another class of examples of crucial input from transfer operators are studies of 
resonances for hyperbolic surfaces of infinite area. For Schottky surfaces (convex 
cocompact surfaces with no cusps and no elliptic points) the standard Markov sym¬ 
bolic dynamics gives rise to transfer operators which represent the Selberg zeta func¬ 
tion. This representation is indispensable for proving the results on the distribution 
of resonances in [12, 23] and the (numerical) investigations of their fine structure 
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in [2, 38]. Our construction of fast/slow systems also applies to non-cofinite Hecke 
triangle groups [31] (forthcoming work will extend it to other Fuchsian groups as 
well). We expect that the arising transfer operators now allow us to extend these 
results on resonances to infinite-area surfaces with cusps. Moreover, the transfer 
operators arising from the slow systems lead to natural conjectures on the residues 
at the resonances. 

In this article we show how to generalize these parallel slow/fast transfer operator 
techniques to accommodate an arbitrary finite-dimensional unitary representation 
(y, x) of the Fuchsian group F under consideration. This provides us with thermo¬ 
dynamic formalism approaches to the Selberg zeta functions for the automorphic 
Laplacian with respect to x (Section 5), and leads to several conjectures on period 
functions for (F, yj-automorphic cusp forms as well as, for infinite-area situations, 
to conjectures on the residues at the resonances (Section 4). 

We restrict ourselves here to cofinite and non-cofinite Hecke triangle groups. How¬ 
ever, it is obvious that these techniques apply (in the same way) to other Fuchsian 
groups as well. The restriction to this class of Fuchsian groups has two reasons. Be¬ 
sides the necessity to keep this article to a reasonable length, Hecke triangle groups 
form a family containing cofinite and non-cofinite Fuchsian groups as well as arith¬ 
metic and non-arithmetic lattices which, in a certain sense, deform into each other. 
Further, the Phillips-Sarnak conjecture [24, 25, 15, 14] states that even Maass cusp 
forms should not exist for generic cofinite Hecke triangle groups, whereas odd Maass 
cusp forms are known to exist in abundance. Both, the conjecture and the results 
in [15, 14], are based on deformation theory. In Section 4.4 we therefore discuss the 
convergence of transfer operators along sequences of Hecke triangle groups. 

It is well-known that any decomposition of the representation yields a corresponding 
factorization of the Selberg zeta functions. In Section 6, we use our results to show 
that this decomposition already happens at the level of transfer operators and that 
the factorization of the zeta functions is merely a shadow of it. 

All Hecke triangle groups commute with the matrix element 

J ePGL2(K). 

Our constructions respect this symmetry in the sense that it is inherited to the 
discretizations, discrete dynamical systems and transfer operators. In Section 7 we 
briefly survey how to extend our results to transfer operator approaches for the 
billiard flow on the triangle surfaces underlying the Hecke triangle surfaces, how it 
can be used to distinguish the odd and even eigenfunctions and to provide separate 
dynamical zeta functions for different boundary conditions. 

2. Preliminaries 

2.1. Hyperbolic geometry. Throughout we use the upper half plane 

HI := {z G C 1 liRz > 0} 

endowed with the well-known hyperbolic Riemannian metric given by the line ele¬ 
ment 

, 9 dzdz 
ds = 


(Imz)2 
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as model for the hyperbolic plane. We identify its geodesic boundary with P^{R) = 
KU {oo}. The group of Riemannian isometries on H is isomorphic to PGL 2 (M), 
whose action on MI extends continuously to P^(K). Its subgroup of orientation¬ 
preserving Riemannian isometries is then PSL 2 (K.), which acts by fractional linear 
transformations. Hence, for [“ ^] £ PSL 2 (R) and z £ HUR. we have 



for cz + d ^ 0 
for cz + d = 0 


and 


.oo = 


for c 0 
for c = 0. 


Q:= 


Q.z = 


0 1 
1 0 

1 


Let 

( 3 ) 

Then 

and 

PGL 2 (R) = PSL 2 (R) U Q. PSL 2 (R). 

Clearly, the action of PGL 2 (R) on H induces an action on the unit tangent bundle 
51HI of H. 


£ PGL 2 


Q.oo = 0, 


2.2. Hecke triangle groups. Let A > 0. The subgroup of PSL 2 (R) generated by 
the two elements 


( 4 ) 



1 

0 


and T\ := 


A 

1 


is called the Hecke triangle group Pa with parameter A. It is Fuchsian (i.e., discrete) 
if and only if A > 2 or A = 2 cos ^ with q £ N> 3 . In the following we only consider 
Fuchsian Hecke triangle groups. A fundamental domain for Fa is given by (see 
Figure 1) 


A < 2 


Hx:={zem\ 1^1 > 1, |Rez| < A/2}. 




Figure 1. Fundamental domain for Fa. 


The side-pairings are provided by the generators (4): the vertical sides {Rez = 
—A/2} and {Re 2; = A/2} are identified via Ta, and the bottom sides {\z\ = 1, Rez < 
0} and {l^l = 1, Rez > 0} via S. The associated orbifold 

:= Pa/H 


is called a Hecke triangle surface. 
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Among the Fuchsian Hecke triangle groups precisely those with parameter A < 2 
are lattices. For A = X{q) = 2cos^ with q € N> 3 , has one cusp (represented 
by oo) and two elliptic points. In the special case q = 3, thus A(q) = 1, the Hecke 
triangle group Fi is just the modular group PSL 2 (Z). The Hecke triangle group r 2 
is called the Theta group. It is conjugate to the projective version of Fo(2). Hence 
X 2 has two cusps (represented by 00 and A/2) and one elliptic point. The groups 
Fa for A > 2 are non-cofinite, the orbifold X\ has one funnel (represented by the 
subset [—A/2,—1] U [l,A/2] of M), one cusp (represented by 00 ) and one elliptic 
point. 

For each Hecke triangle group Fa and each cuspidal point c of Fa, we let Stab(c, Fa) 
denote the stabilizer group of c in Fa. As it is well-known, these are generated by 
a single (parabolic) element, say by Pc- We may choose Poo = T\, and for A = 2, 

2.3. Representations. Let y be a unitary representation of Fa on a finite-dimen¬ 
sional complex inner product space V. Let p GTx be parabolic. We call y singular 
in the cusp represented by p if 

dimker(y(p) — 1 ) > 0 , 

where 1 is the identity operator on V. If y is singular in at least one cusp, then we 
call y singular. Otherwise we say y is regular. Further, we call 

sd(y) := max{dimker(y(p) — 1 ) | p G Fa parabolic} 

the degree of singularity of y. We note that for A 2, the degree of singularity is 
just 

sd(y) = dimker(y(TA) - 1), 

whereas for A = 2 it is 

sd(y) = max{ dimker(y(r 2 ) — 1), dimker(y(P_i) — 1)}. 

2.4. Automorphic functions, cusp forms, and resonances. A function /: H —^ 
V is called {T\,x)-automorphic if 

fil-z) = xil)fiz) 

for all z € H, 7 G Fa- Let C°°{X\;V;x) be the space of smooth {C°°) (FA,y)- 
automorphic functions which are bounded on the fundamental domain Pa, and let 
Cf°{X\; V;x) be its subspace of functions which are compactly supported on Pa. 
We endow Cf° (Aa ; x) with the inner product 

(5) (/i,/ 2 ):=/ (/i(z),/ 2 (^))dvol(z) (/i,/ 2 GCr(AA;H;y)), 

■Irx 

where (•, •) is the inner product on V, and dvol is the hyperbolic volume form. Note 
that y being unitary yields that the definitions of C°° (Aa; ^ x) and (Aa; V; y) 
as well as the inner product (5) do not depend on the specific choice of the funda¬ 
mental domain Pa for Aa. As usual, let 

n ■.= L\Xx\V-x) 

denote the completion of C^{X\]V]x) with respect to the inner product (•,•) 
defined in (5). The Laplace-Beltrami operator 

^ = -y\dl + dl) 
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extends uniquely (Friedrich’s extension) from 

{/ £ C°°{Xx; V\x)\ f s-nd A/ are bounded on T\} 

to a self-adjoint nonnegative definite operator on 'H, which we also denote by A = 
A(rA; x). If / G "H is an eigenfunction of A, say A/ = fif, we branch its eigenvalue 
as /i = s(l — s) and call s its spectral parameter. 

This branching is also useful when considering the resolvent of A, hence the map 

i?(s) := i?(s;F a; x) ■= (A — s(l — s)) Res > 1. 

Here, s(l — s) is understood as the operator s(l — s) idy. This map, as a function of 
s, admits a meromorphic continuation to all of C. Its poles are called resonances. 

For cofinite Hecke triangle groups Fa, A < 2, the cusp (vector) forms in 'H are of 
particular importance, i.e., those L^-eigenfunctions that decay rapidly towards any 
cusp of Xx. More precisely, let 

■.= {v & V \ x{Tx)v = u} 

be the subspace of V which consists of the vectors fixed by the stabilizer group 
Stab(oo,FA) of the cusp oo, and, for A = 2, let 

V-i ■.= {v & V \ x{P-i)v = v} 

be the subspace of V which consists of the vectors fixed by the stabilizer group 
Stab(—I,F a) of the second cusp —1. Then an element f £ V. is called a (Fa,x)- 
cusp form if / is an eigenfunction of A and 

{f{x + iy),v)dx = 0 

for all ?/ > 0 and all v £ t4o, and, for A = 2, also 

1 

[ {f {[-lo]-(.x + iy)) ,v)dx = 0 
Jo 

for all ?/ > 0 and all v £ V-i. We note that these conditions are void if x is 
regular. We further note that for the trivial one-dimensional representation (C, id), 
the (rA,id)-cusp forms are the classical Maass cusp forms. 

2.5. Selberg zeta functions. Let A(Fa) denote the limit set of Fa and i5 := 
dim A (Fa) its Hausdorff dimension. The classical dynamical Selberg zeta function 
for Fa is defined by 

OO 

Zis) := n n , Res > 5, 

£GPL k=0 

where PL denotes the primitive geodesic length spectrum of Xx with multiplicities. 
The Selberg zeta function for the automorphic Laplacian with nontrivial represen¬ 
tation is best stated in its algebraic form, taking advantage of the bijection between 
periodic geodesics on Xx and hyperbolic conjugacy classes in Fa- 

Let g £ Fa. We denote its conjugacy class in Fa by [g\. Let [Fa];i denote the set of 
all conjugacy classes of hyperbolic elements in Fa. Further, if g is not the identity 
element, then there is a maximal n{g) £ N such that g = for some h £ Fa.W e 
call g primitive if n{g) = 1. We let [Fa]p denote the set of all conjugacy classes of 
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primitive hyperbolic elements. If g is hyperbolic, then its norm N{g) is defined to 
be the square of its eigenvalue with larger absolute value. 

The Selberg zeta function for (F^, y) is now defined by 

OO 

zis,x)= n n (i “ 

[5]e[rA]p fc—0 

Through analytic and number theoretic methods, it is known that Z admits a 
meromorphic continuation to all of C. We will provide an alternative proof in 
Section 5 below. Moreover, it is known that the resonances and spectral parameters 
of Fa are contained in the zeros of Z. 

2.6. Special properties of Hecke triangle groups. The Hecke triangle groups 
form a family that satisfies some special properties. They consist not only of a 
mixture of lattices and non-lattices which, in a certain sense, converge to each other, 
but also they mix arithmetic and non-arithmetic lattices. Among the cofinite Hecke 
triangle groups F^ only those for A = 2cos^ with q G {3,4,6} and q = oo, thus 
A = 2, are arithmetic. 

The Phillips-Sarnak conjecture [24, 25] states that for generic lattices one expects 
the space of Maass cusp forms to be finite-dimensional. Hecke triangle groups 
clearly commute with the element Q from (3) or, more obviously from Figure 1, 
with 

7 [-1 0 ] 

J = Q : z 1-^ —z. 

This allows us to separate the odd and the even spectrum. Recall that a Maass 
cusp form / is called odd if f{J.z) = —f{z), and even if f{J.z) = f{z). It is well- 
known that for any cofinite Hecke triangle groups, a Weyl law holds for the odd 
Maass cusp forms. In stark contrast, the results in [15, 14] in combination with the 
Phillips-Sarnak conjecture suggest that even Maass cusp forms generically should 
not exist. 


3. Weighted discretizations and transfer operators 

3.1. Cross sections. We call a subset C of SX\ = Fa\S'IHI a cross section for 
the geodesic flow on X\ if and only if the intersection between any geodesic and 
C is discrete in space and time, and each periodic geodesic intersects C (infinitely 
often). 

The Selberg zeta function only involves the length spectrum of the periodic geodesics, 
and heuristically, the Laplace eigenfunctions are determined by the (statistics of 
the) periodic geodesics only. Therefore, for our applications it is sufficient and even 
crucial to use this relaxed notion of cross section. 

A set of representatives for C is a subset C of 5'EI such that the canonical quotient 
map tt: S’!! —>• SX\ induces a bijection C —>■ C. For any v G SXx let % denote 
the geodesic on Xx determined by 


(6) 


%( 0 ) = V. 
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The first return map of a cross section C is given by 

R: C ^ C, V j!^{t{v)), 
whenever the first return time 

t{v) '■= min{t > 0 | %{t) £ C} 

exists. 

To characterize the set of unit tangent vectors for which the existence of the first 
return time is potentially problematic, let V denote the set of geodesics on 
which converge in the backward or forward time direction to a cusp or a funnel of 
Xx. This means that the geodesic j belongs to V if and only if 7 ( 00 ) or 7 (— 00 ) is 
“in” a cusp or a funnel. Let V/ be the subset of geodesics on Xx which converge 
to a cusp or funnel in the forward time direction. Let T^Vf denote the set of unit 
tangent vectors which determine the geodesics in V/, and let T^V/ := 7 r“^(T^V/). 
For V £ S'!! let jv denote the geodesic on H determined by 7 '( 0 ) = v, and let 
base(u) £ H denote the base point of v. 

Let 

bd := { 7 „(oo) | v £ T^Vf} C R U { 00 } 

be the “boundary part” of the geodesic boundary of H, that is the set of forward 
time endpoints of the geodesics determined by the elements in T^V/. The set 

(pi(R) X bd) U (bdxpi(R)) 

coincides with the set of endpoints ( 7 ( 00 ), 7 (— 00 )) of the geodesics 7 in 7r“^(V). 
Note that if the geodesic 7 is contained in V but not in V/, then the reversely 
oriented geodesic is contained in V/. 

For any subset I of R we set 

1st := / \ bd . 

The cross sections C we constructed in [30] satisfy several special properties. They 
are not intersected at all by the geodesics in V but by each other geodesic infinitely 
often both in backward and forward time direction. Therefore its first return map 
is defined everywhere. They have a set of representatives C" which decomposes into 
finitely many sets C^, a £ A, each one consisting of a certain “fractal-like” set of 
unit tangent vectors whose base points form a vertical geodesic arc on H and all of 
who point into the same half space determined by this geodesic arc. More precisely, 
each is of the form 

C), = {i; € S'!! I Rebase(u) = Xa, 7v(oo) £ la,st, 7v(-oo) £ Rst \ da.st}, 

where = (xa,oo) or = (— 00 , Xa) for some point cCq, £ R which corresponds to 
a cusp or funnel endpoint of F^. 

Via the map r: C —>■ R x A, 

t(v) := ( 7 y(oo),a) for u = 7 r“^(F) fl 

the first return map R: C ^ C induces a discrete dynamical system {D,F) on 

= U ^ 

aeA 
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The special structure of the sets implies that F decomposes into finitely many 
submaps (bijections) of the form 

{la,St n St) X {a} Ip,St X {/3}, (x, a) i-A {ga,p-x, /3), 

where a, P £ A and gct,p is a specific element in T^j. For any v € C'^ there is a first 
future intersection between 7«(K>o) and T.C, say on ga,p.C'p, which completely 
determines these submaps. 

3.2. Weights and weighted transfer operators. We use the representation 
(F, x) of to endow the discrete dynamical system (£>, F) with weights by adding 
to each submap a (local) weight: 

(4.st n g~lp-Ip,st) X {a} Ip,st X {/?}, (x, a) {ga,p-x, P), wt : xiga^p) 

for all a,P £ A. If x G {la,st H g~^p-Ip,st) x {a}, then we say that x has the weight 
wt(x) = xiga^p)- 

Given such a weighted discrete dynamical system {D, F, y), the associated weighted 
transfer operator Cg with parameter s G C is (formally) given by 

^sf{x) := wt(2/)|F'(y)|-V(2/), 

y^F-t{x) 

acting on an appropriate space of functions /:£>—>■ F (to be adapted to the system 
and applications under consideration). 

Due to the special form of our weighted systems, we can deduce a more explicit 
form for the transfer operators. To that end we set 

jsig,x) := (I detgl • (cx + 

for s G C, 5 = [“ ^] G PGL 2 (R) and x G R. Moreover, for a function f: U ^ V 
on some subset of U of P^(R), we define 

xs{g~^)f{x) := tY{ g~^)f{x) := jg{g,x)f{g.x), 

whenever this makes sense. If / is a function defined on sets of the form {Ia,st H 
ga]s-Ip,St) X {a}, then the action only takes place on la,st ^5^ and the bit a 

changes according to its change in the submap. Further, we extend this definition 
to involve the representation by 

as{g) ■= xig~YTY{g): 

hence 

o‘s{g)f{x) = js{g~^,x)x{g~Y {f{g~^-x)) ■ 

Then the (formal) transfer operator Cs becomes 

^sf = ^ ■ 0'siga,p){f ■ 

a,P^A 

where 1e denotes the characteristic function of the set E, and Da,p '■= {Ia,st H 
9a,p-^l3,st) X {q:}- 

If Fa = PSL 2 (Z) and (F, x) is the representation that is induced from the trivial one¬ 
dimensional representation of some finite-index subgroup A of PSL 2 (Z), then our 
weights reproduce the usage of (F, x) in [6, 7, 8]. There this specific representation 
was used to provide a clean bookkeeping tool of the cosets A\PSL 2 (Z) to push 
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the discretizations, transfer operators and period functions from PSL 2 (Z) to A. 
Our usage as weights directly in the subinaps now allows us to accommodate more 
general representations. 

4. The slow systems and automorphic forms 

In the following we recall the cross sections from [30] and state the associated 
weighted discrete dynamical systems. Each of the submaps 

X {a} -)> g.I^t X {/?}, ix,a) i-A ig.x,/3), wt : x(g"^) 

of these systems can be continued to an analytic map on the “analytic hull” of 1st, 
that is the minimal interval / in R such that /\bd coincides with 1st- The associated 
weighted transfer operators are here considered to act on functions defined on the 
analytic hulls. 

If Fa is cofinite and (V, x) is the trivial one-dimensional representation of Fa, then 
we proved in [21, 29, 27] that the highly regular eigenfunctions with eigenvalue 1 of 
the transfer operators are isomorphic to the Maass cusp forms. We expect this to 
hold for more general automorphic cusp forms. Moreover, for the non-cofinite Hecke 
triangle groups, we expect an intimate relation between the transfer operators and 
residues at resonances. 

We divide our considerations into three classes. The first class consists of all cofinite 
Hecke triangle groups with parameter A < 2. These are, in terms of complexity, the 
easiest ones. The cross sections for all these lattices consist of only one component, 
and they are almost identical. The only difference derives from the different sets of 
cuspidal points Fa.oo. The arising discrete dynamical systems are easy stated uni¬ 
formly, even though they decompose into a different number of submaps, depending 
on the order of the elliptic point. 

The second class consists of the Theta group F 2 . The presence of two non-equivalent 
cusps and an elliptic point yields that the set of representatives for the cross section 
decomposes into three components. One immediately sees that only two compo¬ 
nents would be sufficient for a cross section. However, all three components are 
needed for a clean statement of the period functions (i.e., the eigenfunctions of the 
transfer operator) for Maass cusp forms and in particular their regularity condi¬ 
tions. 

After surveying this result and its existing and expected generalizations to more 
general automorphic cusp forms, we investigate the reduced cross section for r 2 , 
the relation between the arising transfer operators from the two cross sections, 
and provide the characterization of Maass cusp forms in this second system. More 
importantly, we will see that this reduced cross section is very similar to the cross 
sections for the non-cofinite Hecke triangle groups, our third class. As we will 
see, this similarity allows us to prove a convergence of transfer operators along a 
sequence of Hecke triangle groups. 

Throughout we omit some dependencies from the notation, and we will use (D, F) 
to denote any discrete dynamical system. 

4.1. Hecke triangle groups with parameter A < 2. Let F = Fa be a cofinite 
Hecke triangle group with parameter A = X{q) = 2cos^, q £ N> 3 . The set of 
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representatives C in [30] for the cross section G of the geodesic flow on r\H is 
C := {v £ S'!! I Rebase(i;) = 0, 7„(— oo) £ (—oo,0)st, 7v(oo) £ (0,oo)st}- 
For fc £ Z let 

fffe := {{TS)’^Sy\ 

The induced discrete dynamical system {D,F) is defined on £> = (0,oo)st and 
decomposes, as it can be read off from Figure 2, into the q — 1 weighted submaps 



g^-C' 




Figure 2. Cross section and future intersections for Fa, A < 2. 


( 5 fe ^■oo)st-t (0,oo)st, x^gk-x, wt : ^), 

where A: = 1,..., g — 1. The associated transfer operator Cg.x Fct(IR>o; V) 

via 

q-l 

k=l 


Theorem 4.1 ([21]). Let {V,x) = (C, id) be the trivial one-dimensional repre¬ 
sentation of Fa and let s £ C, Res £ (0,1). Then the space of Maass cusp 
forms for Fa with spectral parameter s is isomorphic to the space of eigenfunctions 
f e C-(K>o ; C) of Ls^x eigenvalue 1 for which the map 


f on R>o 

-Ts{S)f ouR<o 


extends smoothly to all o/R. If u is a Maass cusp forms with spectral parameter s, 
then the associated eigenfunction f £ C"^(M>o;C) is given by 



[u,R(t, •)"], 


where the integration is along any path in HI from 0 to ioo, and R{t, z) := Im 
and 

du dv 

= — • vdz + u ■ T^az. 
oz oz 


[u,v] 



In number theoretical terms, the space of eigenfunctions of ia Theorem 4.1 
constitutes period functions for the Maass cusp forms. Let u be a Maass cusp form 
with corresponding period function /. The condition that / is a 1-eigenfunction of 
£s,x corresponds to u being an eigenfunction of the Laplace-Beltrami operator, and 
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the regularity requirement / G C'‘^(R>o;C) corresponds to u being real-analytic. 
To model the property that u is rapidly decaying towards the cusp on the side 
of period functions, we first need to extend the period functions to almost all of 
M (actually of P^(R)). In Figure 2 we see that S.C' is “opposite” to C in the 
sense that S.C'VJC is disjoint and almost coincides with the complete unit tangent 
spaces at i]R>o. Further, S.C is conversely oriented to C. Therefore, / is extended 
by —Ts{S)f in (7). Now the cusp is represented by 0, for which reason we need 
to request that (7) is smooth at 0. Since oo represents the same cusp, there is no 
additional requirement at oo. 

The proof of Theorem 4.1, for which we refer to [21] or [27], makes crucial use of the 
characterization of Maass cusp forms in parabolic cohomology by Bruggeman, Lewis 
and Zagier [3]. Such a characterization is not (yet) available for (Fa, x)-automorphic 
cusp forms with nontrivial representation. However, for representations of PSL 2 (Z) 
that are induced from the trivial one-dimensional representation of a hnite index 
subgroup, Deitmar and Hilgert [8] proved an analogue of Theorem 4.1 using hyper¬ 
function theory. We expect that results analogous to Theorem 4.1 are true in much 
more generality. 

Conjecture 4.2. Let (V, x) be any unitary finite-dimensional representation of Fa, 
and Res G (0,1). Then there is a bijection between the (Fa, x)-automorphic cusp 
forms with spectral parameter s and the 1-eigenfunctions of the transfer operator 
same regularity as in Theorem 4.1. 

4.2. The Theta group. Let F := r 2 be the Theta group. 


4.2.1. The original system. The cross section C in [30] for F is given by 


C :=C'uauC', 


where 

C' := {v G S'!! 1 Rebase(z;) = -1, 7 „(-oo) G (-oo,-l)st, 7 «(oo) G (-l,oo)st}, 

Cft := {u G OT 1 Rebase(u) = 1, 7 „(-oo) G (l,oo)st, lv[co) G (-oo,l)st}, and 

C' := {v G S'!! 1 Rebase(i;) = 0, 7 „(-oo) G (-oo,0)st, Jv{oo) G (0, oo)st}. 


Let 


fci := T, 


k2 := T-^S = 


2 

-1 




-1 

0 ’ 


and ki := S. 


From Figure 3 it can be read off that the induced discrete dynamical system (D, F) 
is defined on 


D := ((-1, oo)st X {a}) U ((-oo, l)st x {&}) U ((0, oo)st x {c}) 


and decomposes into the submaps 

(-l,0)st X {a} -)■ (-l,oo)st X {a}, 
(0,oo)st X {a} (0,oo)st X {c}, 
(-oo,0)st X {6} (0,oo)st X {c}, 
(0, l)st X {6} (-00, l)st X {6}, 

(0, l)st X {c} (-00, l)st X {6}, 


{x,a) 


{h 

.x,a), 

wt 

to 1 


(x,a) 


(x,: 

c), 

wt 

: X(l)t 


{x,b) 


(fc4 

.x,c), 

wt 

: X(.k4 


{x,b) 

1-^ 

{ks 

.x,b), 

wt 

■ Xiks 


(x,c) 


{ks 

.x,b), 

wt 

■ Xiks 
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fci.c: 


Figure 3. Cross section and future intersections for r2. 


(l,oo)st X {c}-)■ (-l,oo)st X {a}, {x,c) ^ .x,a), wt : x(fci ^)- 

For each / G Fct(D; V) we let 

fa • f * 1( —l,oo)stX{a}; fb • f ‘ 1( —oo,l)st x{I)} and ff. — f • l(o,oo)st X {c} • 

We identify / with the vector 

^fa 
fb 
\fc 

and then (—l,oo)st x {a} with (—l,oo)st, (—oo, l)st x {6} with (—oo,l)st, and 
(0, oo)st X {c} with (0, oo)st- The associated transfer operator parameter 

s G C and weight x is then represented by 


r — 

^s,x — 


^as{k2) 0 as(k^^)'' 

0 asiks) asiks) 

q;s(1) as{k4) 0 


The following theorem is a special case of the very general results in [27]. 

Theorem 4.3. Let {V,x) = (C,id) be the trivial one-dimensional representation 
ofT, and let s € C, 1 > Res > 0. Then the Maass cusp forms u for T with spectral 
parameter s are isomorphic to the function vectors f = {fa, fb, fc)^ such that 

/aGC-((-l,oo);C), /fcGC“((-oo,l);C), and /, G C“((0,oo);C), 

and f = Cg^xf ’ map 

j fc on (0, oo) 

\-'rs(<S')/c on (-00,0) 

extends smoothly to K, and the map 

\fa on (-l,oo) 

\-Tg{T-^)fh on (-00,-1) 
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extends smoothly to 
rem 4-1 for notation) 


The isomorphism from u to f is given by (see Theo- 

/ ICO pico 

fb{t) = -j [u,R{t,-y], 


and 


r 

m = / 
^0 


i,R{t,-y 


As in Section 4.1 we again expect that Theorem 4.3 can be generalized to arbitrary 
(r, x)-automorphic cusp forms. 

4.2.2. The reduced system. Figure 3 shows that already 7r(C^ U C^) constitutes a 
cross section. The induced discrete dynamical system on 

D := ((-l,oo)st X {a}) U ((-oo,l)st x {6}) 
is given by the submaps 


( t 5 t))st X {ci} y ( 1, Oo)st 

X {a}, 

(x, a) !->■ (k 2 .x, a), 

wt 

X{kfy 

(0, l)st X {a} -)■ (-00, l)st 

X {b}, 

(x,a) !-)> (fca.x, 6), 

wt 

Xikfy 

(l,oo)st X {a} -)■ (-l,oo)st 

X {a}, 

(x, a) 1 —>■ {kf^.x, a), 

wt 

Xiki) 

OO, -l)st X {b} -)■ (-00, l)st 

X {b}, 

{x,b) !-)> {ki.x,b), 

wt 

xikfy 

(-l,0)st X {b} (-l,oo)st 

X {a}, 

{x, b) !->• {k 2 .x, a), 

wt 

to 1 

(0, l)st X {b} -)■ (-00, l)st 

X {b}, 

{x,b) !-)> {k 3 .x,b), 

wt 

xikfy 


If we represent / £ Fct(Zl; V) by the function vector 

^faf 

h 


then the associated transfer operator is represented by 

«.s(fcr^) + otsih) 
asiky a. 




Ols{k2) 

(fci) + as{ky 


(r) 

Even though the transfer operator derives from a smaller cross section than 
£s,xi it® highly regular 1-eigenfunction vectors characterize the Maass cusp forms 
for F in the case (V, %) = (C, id) as we will show in the following. 

Lemma 4.4. The map 



is an isomorphism between the 1-eigenfunction vectors of and the 1-eigenfunction 
vectors of Ts^x- 

Proof If / = {fa, fb, fcV is a 1-eigenfunction of Cs,x, then 

fc = as{l)fa + asikyft. 

□ 
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The combination of Theorem 4.3 with Lemma 4.4 yields the following Corollary. 

Corollary 4.5. Let (V,x) = (C,id) and Res € (0,1). Then the Maass cusp 
forms for T with spectral parameter s are isomorphic to the 1-eigenfunction vectors 
f = {fa,fbV of which satisfy 

/a GC"" ((-l,oo);C) and A G C"" ((-oo, 1); C), 
and for which the map 

ifa + TsiS)fb on(0,oo) 
l-Ts(S)fa-fb on (- 00 , 0 ) 
extends smoothly to M, and the map 

\fa on {-l,oo) 

\-Ts{T~^)fb on (-00,-1) 

extends smoothly to P^(R). 

The isomorphism in Corollary 4.5 is given as in Theorem 4.3. 

4.3. Non-cofinite Hecke triangle groups. Let T be a non-cofinite Hecke triangle 
group, thus T := Ta for some A > 2. The set of representatives C for the cross 
section C from [31] decomposes into two components 

c :=c';ua 

which are 

C'^ := {v G S'!! I Rebase(u) = —1, 7 „(—oo) G Mst, 7 v(oo) G (—l,oo)st}, and 
Cb '■= {v G OT I Rebase(u) = 1, jy(-oo) G Mst, 7 «(oo) G (-oo, l)st}. 

The induced discrete dynamical system (D, F) can be read off from Figure 4, where 




a 


CL 


oi-CL 


1-A -1 



1 A-1 


Figure 4. Cross section and future intersections for Fa, A > 2. 


ai := T, 02 ■■= T-^S = 

Thus, it is defined on 

D := ((-l,oo)st X {a}) U ((-oo,l)st x {6}) 
and given by the weighted submaps 

(-l,0)st X {a}->■ (-l,oo)st X {a}, (x,a) (a 2 .x,a), wt : x(a^^). 


and 03 := TS = 


A -1 
1 0 
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(0, l)st 

X 

{o} 


(- 

-cx: 

, l)st 

X 

{b}, 

[x,a) 


(03. 

.x,b), 

wt 


(-1 -1- A, oo)st 

X 

{o} 


(- 

-1, 

00)st 

X 

{«}, 

[x,a) 


(or 

\x,a). 

wt 

: X(ai), 

(- 00 , 1 - A)st 

X 

{b} 


(- 

-cx: 

; l)st 

X 

{b}, 

{x,b) 


(oi. 

.x,b), 

wt 


(-l,0)st 

X 

{b} 


(- 

-1, 

Oo)st 

X 

{«}, 

{x,b) 


(02. 

.x,a), 

wt 

■ 

(0, l)5t 

X 

{&} 


(- 

-cx: 

; l)st 

X 

{b}, 

{x,b) 


(03. 

.x,b), 

wt 



For / S Fct(£>; V) we set 

fl — I ' 1( — l,oo)st x{a} Rnd f2 — f * 1( — oo,l)st X {6} : 

and identify / with 

h 

/2 

as well as (—l,oo)st x {a} with (— l,oo)st, and (—oo,l)st x {6} with (—oo,l)st- 
The associated transfer operator parameter s € C and weight x is then 

represented by 

as(a2) + as(a^^) a2(a2) \ 

0 ^( 03 ) as(ai) + 05 ( 03 )/' 



In view of Theorems 4.1 and 4.3 we expect the following significance of the 1- 
eigenfunction vectors of /ls,x- This conjecture is also supported by Theorem 5.6 
below. 

Conjecture 4.6. The (sufficiently regular) 1-eigenfunction vectors of Cs,x deter¬ 
mine the residues at the resonance s. 


4.4. Convergence along sequences of Hecke triangle groups. The Phillips- 
Sarnak conjecture on the (non-)existence of Maass cusp forms is motivated by 
considerations using deformation theory along families of Fuchsian lattices. Also 
the results in [15, 14] are based on such a deformation theory. With this in mind, it 
is natural to ask how transfer operators behave along families of Fuchsian groups. 

We recall from Section 4.3 that the elements in the discrete dynamical system for 
Fa with A > 2 are 


ai(A) 


1 A 
0 1 ’ 


a2(A) 


A 1 
-1 0 ’ 


03 (A) 


A -1 
1 0 


and the transfer operator is given by 

^Q;s(a2(A)) -I- 05(01 (A)“^) 


rW = 
^s,x 


asiaaiX)) 


as{a2{X)) 

as{ai{X)) + asiasiX)) 


In Section 4.2.2 we see that for A = 2 we have ai(2) = ^ 1 , 02 ( 2 ) = k2,a3{2) = 
and that the transfer operator for the reduced system for the Theta group F 2 is 
just 


Theorem 4.7. We have the convergence 

in operator norm for any choice of norm on Fct((—1, 00 ); V) x Fct((— 00 ,1); V). 
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We remark that Theorem 4.7 is very sensitive to the choice of the cross sections and 
their sets of representatives. Similar results hold for the transfer operators along 
the family of cofinite Hecke triangle groups if one uses a different cross section for 
the Theta group. The details will be discussed in the forthcoming Master thesis of 
A. Adam (Gottingen). 

5. The fast systems and the Selberg zeta functions 

In this section we discuss how to represent the Selberg zeta function Z(s, x) for 
(with the representation (V,x)) eis a Fredholm determinant of a family of transfer 
operators arising from a discretization of the geodesic flow on X\. 

Each of the slow discrete dynamical systems in Section 4 is expanding but none 
uniformly. This has the consequence that their associated transfer operators are 
not nuclear (on no nonzero Banach space) and hence do not admit Fredholm deter¬ 
minants. The reason for this non-uniformity in the expansion rate is the presence 
of submaps of the form 

Ja X {a}X {a}, [x,a) ^ {g.x,a), 

where Ja is a subset of la and g is parabolic or elliptic. To overcome this problem 
we induce the slow discrete dynamical systems on these submaps. The outcome 
are discrete dynamical systems which branch into infinitely, but countably many 
submaps that are uniformly expanding. We call these systems fast. 

The definition of transfer operator from Section 3.2 clearly carries over to fast 
discrete dynamical systems. But now the associated transfer operators involve 
infinitely many terms which makes necessary a discussion of their convergence and 
domains of definition. We show below that they constitute nuclear operators of 
order zero on direct sums of Banach spaces (with supremum norm) of the form 

B{C]V) := {/: C —>■ E continuous, f\c holomorphic}, 

where C is an open bounded disk in P^{C). The action Ts obviously extends to 
B{C; V). We continue to denote each arising discrete dynamical system by {D,F). 

5.1. Hecke triangle groups with parameter A < 2. Let F := Fa be a cofinite 
Hecke triangle group with parameter A < 2. Recall the elements gk & T, k = 
1,..., g— 1, from Section 4.1. The fast discrete dynamical system {D, F) is defined 
on D = (0, oo)st and given by the weighted submaps 

(0,oo)st, x^gk-x, wt : x(fffc 

for fc = 2,..., g — 2, and, for n G N, 

-t (0, A)st, X g'^.x, wt : xigi'^), 

Q,oo^ , x^g^_^.x, ■ xi 9 q-i)- 

We identify / € Fct{D; C) with the function vector / = (/i, /r, where 

fl • f l(A,oo)st’ f ■ f ^(l/A,A)st; ^-nd fq—1 • f f(0,l/A)st' 
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Then the associated transfer operator with parameter s € C is formally given by 
the matrix 

A 


/ 


9-2 


0 

E ^sigk) 

k=2 

9-2 

E 

n€N 

E 

ois{gi) 

E Ots{gk) 

E 

raGN 


k=2 

9-2 

neN 

E 

\raGN 

ois{gi) 

E a.s{gk) 

k=2 

0 


r — 


To find a domain of definition for Cs^x which it becomes a nuclear operator, we 
need to work with neighborhoods of oo in P^(C). To avoid dealing with changes of 
charts we conjugate the group T, the discretization, the discrete dynamical system 
and Cs,x with 


r = 


1 


1 -1 


72 [1 1 . ■ 


Then A := TFT ^ is the conjugate lattice, the conjugate discrete dynamical system 
is {E, G) defined on 

E :={-!, lU 

and given by the submaps from above, where each is changed to 

hk ■= TgkT~^ 

and the weight is determined by the representation 

g:A^U{V), g{h) ■.= x{T-^hT). 


Let 

and 


I3s{h):=g{h ^)Ts{h) 
A- 1 


El := T.(A, oo) = 

1 


Er := r. 


A’' 


A = - 


A + 1’^ 

A-1 A-1 


E,_i :=r. 0,- = -1,- 


A + 1’ A+1 
A-1 


A + 1 


The conjugate transfer operator 

/ 


r — 

^s,r] — 


9-2 


0 


E Ps{hk) E W-i) 


k=2 

9-2 


neN 


E Ps{K) E Ps{hk) E Ps{K-i) 


k—2 

E E /37^fc) 

\neN k—2 


neN 


is formally acting on the function vectors 

/ = 


fi: El 

fr- Er^V 

fq-1 ■ Eq-i VJ 
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As shown in [21, Proposition 4.2] there exist open bounded disks £i,£r,£q-i C C 
such that 

(i) Ej C £j for j e {1, r, q - 1}, 

(ii) J.£r = £r and J.£i = £q-i, where J = TQT~^ = [ V i ] > 

(iii) for fc = 2,..., g — 2 we have .£i C h^^.£r C £r and .£q-i C £j., 

(iv) for n € N we have h^'^.£r ^ £i and C 

(v) for n G N we have h~^^.£i C £q-i and h~^^.£r ^ £q-i^ 

(vi) for all 2 ; G we have Rez > —1, 

(vii) for all z G £q-i we have 1 > Rez, and 
(viii) for all z G fr we have 1 > Rez > — 1. 

Let 

B{£- V):=B{£i-V)® B{£r; V) © ; V). 

Theorem 5.1. (i) PorRes > the transfer operator Cg^r] 'is an operator on the 

Banach spaee B{£;V) and as such nuclear of order 0. 

(ii) The map s 1 —>■ Cg^q extends meromorphically to all o/C. We also use Cg^q to 
denote this extension. The poles are all simple and contained in ^(1 — No). 
For each pole sq, there is a neighborhood U of sq such that Cg^q is of the form 

^s.q — As © Bg 

S- So 

where the operators Ag and Bg are holomorphic on U, and Ag is of rank at 
most 4 sd(77). 

Proof. The proof of (i) is an easy adaption of the corresponding statement in [21] 
for the trivial one-dimensional representation. For (ii) we need to show that the 
maps 


{Res > |} 


{operators B{£q-i;V) —>■ B{£i;V)} 

s 





riGN 

(Res > |} 

-)■ 

(operators B{£q-i;V) —>■ B(£r;V)} 

s 

i-A- 

E Ps{h'^-i) 



nGN 

{Res> 


(operators B{£i-,V) —>■ B{£r]V)} 

s 

1 -^ 

E Ps{hf) 



neN 

(Res > 

—> 

(operators B{£i]V) —>• B{£q-i]V)} 

s 

!-)■ 

E /3s(^i) 


riGN 



extend to meromorphic functions on C with values in nuclear operators of order 0 
and with poles as claimed. We provide these proofs for the latter two maps. The 
proofs for the former two maps are analogous. We start with a diagonalization. 

Since ri{hi) is a unitary operator on V, there exists an orthonormal basis of V with 
respect to which ri{hi) is represented by a (unitary) diagonal matrix, say 


diag (e2™i,...,e2™‘^) 
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with ai,... ,ad S M and d = dim]/. The degree of singularity sd(r]) is then the 
number of integral aj in this representation. We use the same basis of V to represent 
any f £ B{£k',V) (fc G { 1 , r, g — 1 }) as a vector of component functions 


//i 


\fd/ 


: £k ^ C''. 


In these coordinates, the operator on the right hand side in the definition of 
and becomes 


diag 


,2nina^T,{h^] 


'^riGN 


Note that here t, = rr. Let 


neN 


L. :=^ 




riGN 

with a G R. Then it suffices to show that ior i G {r,q — 1} and a G R., the maps 

{Res > 5 } —>■ (operators B{£i;C) B{£f,C)} 

s I—^ Lg 

admit entire extensions if a G R. \ Z, and, if a G Z, meromorphic extensions to all 
of C with simple poles all of which are contained in i(l — No). 

Note that 1 G is the fixed point of the parabolic element For M G Nq 

let Pm ■ B{£i-C) —>■ B{£i\C.) be the operator which subtracts from a function its 
Taylor polynomial of degree M centered at 1, thus 

Pm{ 9 ){z) := gM{z) ■= g{z) - ^ ^ {z - I)''. 


fe =0 


We use the operator Pm to write Lg as a sum of two operators: 

(8) Lg = Lg o (^1 — Pm) P Lg o Pm- 

We start by investigating the first term on the right hand side of ( 8 ). For n G N 
we have 


and hence 


h" = - 
^ 2 


hj -”.2 = l-- [n 


2 + nA —n\ 
nX 2 — nX 


A(1 - z) 


Let Re s > ^, g G B{£i;C) and z G £rdl £q-i. Then 

{Lg o (1 - PM))giz) = Lg{g - gM){z) = ^ ^^Lg{{z - 1)'=) 


M 


= E(-i) 






A:! A2«+^(l - z)‘^^ 


nGN 


A(l- z) 


- 2 s-k 
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We recall the Lerch zeta function 


C{s,a,w) := ^ 

n—0 


g27rma 

(n + w)® ’ 


defined for Res > 1, a G R and w £ C \ (—No). For a G Z, this is just the Hurwitz 
zeta function. A classical Riemann method shows that the Lerch zeta function 
extends meromorphically to all of C in the s-variable. For that one considers the 
contour integral 


/(s, a, w) 


1 f ^ 

- / - ^ ■ dz, 

2Tri Jc 6 ^+ 2 ^^“ 


where C is a path which begins at —oo, goes once around the origin in positive 
direction, and returns to —oo such that it does not enclose any point in 27rz(—a + Z) 
other than (possibly) 0. Then I{s,a,w) is entire in s, and 


C(s, a, tc) = F(s — l)/(s, a, tc). 


Hence, for a G Z, the map s i—>■ ((s,a,w) extends to a meromorphic function on 
C with only a simple pole at s = 1. For a ^ Z, the extension of s i—>■ ({s,a,w) is 
entire. 


The previous calculation now shows that 

M 

Ls{g-9 m){z) = 

k^O 

From 1 > Rez it follows that Re ^1 + ^ Thus, the properties of the 

Lerch zeta function imply that for a G Z the map Ls{g — gM){z) extends meromor¬ 
phically in the s-variable to all of C with simple poles at s = (1 —fc)/2, fc = 0,..., M. 
For a the map La{g — gM){z) extends to an entire function. In both cases, the 
extension of Lg o (1 — Pm) is nuclear of order 0 as a finite rank operator. 

For the investigation of the second term of (8) we fix t > 0 such that the ball Rt(l) 
in C with radius t around 1 is contained in Ei. Let g G B{£i; C) and M G N. There 
exists C G R such that for all z G 

\9m{z)\<C\z-1\^+\ 

Further, since hi is parabolic with fixed point 1, we have 


^ ^ ^ :C 2s + fc,a,l + 


fc! A2®+fc(l-z)2 


A(1 - z) 


lim h-^.z = 1 

n—^oo 

for every z G C. Thus, for each z € £r U £q-i and for some uq = '^^{z) G N, we 
have 


\{Ls o PM)g{z)\ = 


< 


E 

nGN 

E 

n<no 


(nA(l — z) + 2)" 


g27rma22s 


:5M(fci ".z) 


(nA(l — z) -I- 2)^ 


:5m(/ii".z) 


E 

n>no 


22ReSg7r| Im s\ 


|nA(l - z) -I- 2|2Res-|-M-|-l ■ 
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Hence, {Ls o PM)g{z) converges for Res > — "y- The operator Lg o Pm is nuclear 
of order 0 since Pm is bounded. 

Obviously, these arguments apply to each M S N, and hence Lg extends meromor- 
phically to C with simple poles at s = (1 — k)/2, k G Nq, for a £ Z, and it extends 
holomorphically to C for a ^ Z. This proves (ii). □ 


An immediate consequence of Theorem 5.1 is that the Fredholm determinant 

det(l - Cg^r,) 

exists and defines a holomorphic map on Re s > i. For Re s sufficiently large (here, 
Res > 1), it is given by 

det(l - Lg^r,) = exp ( “ X! ^ TcZ^.r,) ■ 

rieN 

To show that it equals the Selberg zeta function Z{s,r]) we need the following 
lemma. 


Lemma 5.2. Let h G A and letC be an open bounded subset ofC such that h.C C C 
and sueh that the action of h on C has a single fixed point. Consider the operator 
Tg{h~^) = tY{ h~^) as acting on the Banach space B{C; V). Then 

Tr{l3g{h-Y) = 


Proof. For T^{h ^): B{C;C) —>■ B{C;C) we have (see [33]) 


TrT^ih-Y = 


Nih)- 


1-N{h)-^' 

Since g{h) is a unitary operator on V, we find a basis of V with respect to which 
r]{h) is represented by a diagonal matrix, say by 

diag(ai, ...,ad) 

with oi,... ,ad G C, d = dimH. We use the same basis of V to define an isomor¬ 
phism (of Banach spaces) 

d 

i?(C;H)-0i?(C;C). 

1=1 

Under this isomorphism, the operator tY (h) acts diagonally, i.e. 

tY {h-Y = 0pf(/i”^)- 
1=1 

Hence 


vWrYih ^)=0ajTf(/i Y 


1=1 


and 


TTgih)TYih i)=^a,Trrf(h ') = ^ E 


1=1 


1=1 


1 - N{hy 


□ 
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Theorem 5.3. (i) For Res > 1 we have Z{s,r]) = det(l — iCs,??)- 

(ii) The Fredholm determinant det(l — Cg^n) extends to a meromorphic function 
on C whose poles are contained in ^(1 — Nq). The order of a pole is at most 

4sd(?7). 


Proof. We start with the proof of (i). Let n € N and suppose that h = sq ... s„_i 
is a word over the alphabet 

(9) {hT,h2,...,hg-2,h'^_,\mGn}. 

We call h reduced if it does not contain a subword of the form or 

mi , m 2 € N. We say that h is regular if both h and hh are reduced. The length of 

h is n. 


Since the semigroup iL in A generated by (9) is free (see [21]), each element h G FI 
corresponds to at most one word in this alphabet. For this reason, we will identify 
the elements in iJ with their corresponding words. 

Let Pn denote the set of regular words of length n. An immediate consequence of 
[21, Proposition 4.1] is the identity 

Tr£,,^ = ^ TvPgih). 
hePn 


Further, let g € A be hyperbolic. By [21], the conjugacy class [g] contains at least 
one regular representative h G Tl. Let /iq be a primitive hyperbolic element such 
that hg = h for some n £ N. Then hg G H. The word length w{h) of h and the 
word length w{ho) of hg are independent of the chosen representatives. We define 
w{g) '■= w{h) to be the word length of g, and p{g) '■= w{ho) the primitive word 
length of g. Then there are exactly p{g) representatives of [g] in Pw{g)- Note that 
nig) = w{g)/p{g). 

For Re s > 1 we now have 


In Z(s) 


OO 

[5]^[r]p ^‘—0 

[ff]G[r]pfei > 

1 N{g) ^ ^ / I 

[s]e[r];, 

V- pig) Nig)-^ ^ ^ 

[9]^W\h 

1 Niji) ^ 

-E- E 1 _„(!.)-■ ‘--iw 

™ = 1 h-^ePn, ^ ^ 

OO ^ 

Tr/3s(h) 

^' w ^' 

W=1 hePm 


-V - Tr£“ 


w—1 
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= In (^det(l - Cs,r,)j ■ 

This proves (i). Statement (ii) follows now immediately from Theorem 5.1(ii). □ 

Corollary 5.4. For regular t], the map s i— >■ £s ,?7 o,nd the Fredholm determinant 
det(l — Cs^n) extend to entire functions. 

For PSL 2 (Z) it is known (see [5, 16]) that the 1-eigenfunctions of the “slow” trans¬ 
fer operators from Section 4.1 and the “fast” transfer operators developed in this 
section are isomorphic. Their proof takes advantage of special properties of the 
modular group. However, by geometric consideration we expect that the same 
relation holds for general Hecke triangle groups. 

5.2. The Theta group. Let T := r 2 be the Theta group. Recall the elements 
ki, ^ 2 , fca G r from Section 4.2. The first step from the original slow system for F in 
Section 4.2.1 towards a fast system has already been made in Section 4.2.2, where 
we eliminated the acting elliptic element S and the action by 1 G F. The additional 
induction on the parabolic elements gives rise to the fast discrete dynamical system 
{D,F) which is defined on 


D := ((-1, oo)st X {a}) U ((-oo, l)st x {6}) 


~ ((0)1)81 X {a}) U (( l, 0 )st X {a}) 

U ((l,oo)st X {a}) 


U ((-l, 0 )st X {b}) U ((-oo,-l)st 

X {6}) U (( 0 ,l)st X 

{b}) 

and determined by the weighted submaps 



(0, l)st X {a} (-00, l)st x {b}, 

(x, a) {ks.x, b), 

wt : x(fc3'^) 

(-l,0)st X {b} (-l,oo)st X {a}, 

(x, b) !->• {k 2 .x, a), 

wt : x{k2^) 

and for n G N 



(j (2 kf^.ooj X {a} —>■ ( 0 , oo)st x {a}. 

(x, a) 1—>■ (ktf.x, a), 

wt: 

(fc^(-l), fcr^(-i)),, X {«} ^ (-1, i)st X {«}, 

(x, a) 1—>■ (kf^.x, a 

), wt:x(fci) 

(fc-(”+i).l, X {b} ^ (-1, l),t X {6}, 

(x,6) !-)> (fc^x,6). 

wt : x(fcr") 

(^k^'^.oo, ^ X {6} (- 0 O, 0 )st x {b}, 

(x,6) !-)> (/c3.x,6). 

wt : xiks^) 

Note that 

= (-l,0)st X {a}, 



neN 

and analogously for the other infinite branches. As before, we identify / G Fct(Il; V) 
with the function vector (/i,..., /e)^, where 


/i ■— / • l(-l,0),tX{a} 

f2 ■= f ■ l(0,l)atX{a} 

fs ■= f ‘ l(l,oo)stX{a} 

/4 ■ / ■ 1(—oo,—l)stX{&} 

/s := / • l(-l,0),tx{t)} 

fe ■= f ■ l(o,i).tx{b}: 
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and we may omit the x{a} and x{6}. Then the associated transfer operator is 
(formally) represented by the matrix 




E “44 ") 


Cis{k2) 










E «4fcr”) 


ag{k2) 


raGN 






E Oigik^) 




a.g{k2) 


raGN 

Ctgiks) 




E Mm 








asih) 


E Oig{k^) 


E Mm 




n£N 


riGN 


asih) 


E Oig{k^) 










For a,b G P^, a b, let B{a, b) denote the open ball in P^(C) which passes through 
a and b. If a > 6, then B{a, b) is understood to contain oo. 

Let 

Let 

6 

:=0i?(P,;F). 

j=i 

Analogously to Theorems 5.1 and 5.3 one shows the following relations between 
the transfer operator family and the Selberg zeta functions. Note that the order of 
poles differs from that in Theorems 5.1 and 5.3. 

Theorem 5.5. (i) For Res > the transfer operator Cs,x *5 operator on 

B{'D; V) and as such nuclear of order 0. 

(ii) For Res > 1, we have Z{s,x) = det(l — Ls^yf)- 

(iii) The map s i-A extends meromorphically to all of C. The poles are all 
simple and contained m ^(1 — Nq). For each pole Sq, there is a neighborhood 
U of So such that the meromorphic extension is of the form 

Fs,'y — wds “t“ Bs^ 

s - So 

where the operators As and Bg are holomorphic on U, and Ag is of rank at 
most 6sd(x). 

(iv) The Fredholm determinant det(l — Ts,x) extends to a meromorphic function 
on C whose poles are contained in ^(1 — No). The order of a pole is at most 
6sd(x). 

5.3. Non-cofinite Hecke triangle groups. Let T := F;^ be a non-cofinite Hecke 
triangle group, thus A > 2. Recall the elements Oi, 02,03 G F from Section 4.3. The 
fast discrete dynamical system {D, F) for F is defined on 

D := ((-1, oo)st X {a}) U ((-00, l)st x {b}) 





SYMBOLIC DYNAMICS AND THE AUTOMORPHIC LAPLACIAN 


27 


and given by the weighted submaps 
(-l,0)st X {a} (-l,oo)st x {a}, 
(0, l)st X {a} -)■ (- 00 , l)st X {6}, 
(-l,0)st X {&} -?■ (-l,oo)st X {a}, 
(0, l)st X {6} -)■ (- 00 , l)st X {&}, 


{x,a) {a 2 .x,a), 

wt : x(a 2 ^ 

{x,a) i-A ( 03 .T, 6 ), 

wt : x(a 3 ^ 

{x, b) i-A {a 2 .x, a), 

wt : x{af^ 

{x,b) i-A ( 03 .T, 6 ), 

wt : x(a 3 ^ 


), 

), 

), 

), 


and, for n G N, 


(—1 + nA, —1 + (n + l)A)st x {a} — >■ (—1, —1 + A)st x {a}, 

{x, a) i-A a), wt : x{ai ), 

(1 - (n + 1)A, 1 - nA)st x {b} {1 - X, l)st x {6}, 

(x, b) ^ {a^.x, b), wt : x(a^”). 


The associated transfer operator with parameter s G C is represented by the matrix 



(as{a2) 

E as(ai”) 


Q:s(a2)\ 






r — 

— 

O^s{0'2) 

asias) 



as{a2) 

asias) 


^asias) 


E ois{af) 

n€N 

asias) 


acting formally on the function vectors 

/ /i: \ 

/2 : (~1 + A, Oo)st — >■ V 
/s: (~oo, 1 — A)st — >■ V 
V /4: (-l,l)st^ y 


/ = 


Let 


Pi:=B(-l,l), V2:=B 


5A-4 
6 ’ 


and 2?3 := B 


A 4-5A 
2 ’ 6 


and set 

B{V- V) := B{Vi,V) 0 B{V2]V) 0 B{V3] V) 0 B{Vi;V). 


Analogously to Theorems 5.1 and 5.3 the following results are shown. 

Theorem 5.6. (i) For Res > the transfer operator Cs,x defines a nuclear 

operator on B{'D] V) of order 0. The map s 1 —>■ Ts.x extends meromorphically 
to all of <C with possible poles located at s = {1 — k)/2, k G Nq. All poles are 
simple. For each pole sq, there is a neighborhood U of sq such that Cs^x ** 
the form 

Bs X — A-s 0 Bs^ 

s - So 

where the operators As and Bs are holomorphic on U, and As is of rank at 
most 2 sd(x). 

(ii) For Res > max(dimA(r), i) we have Z{s,x) = det(l — Cs,x)- Moreover, the 
Fredholm determinant s i-A det(l — Cs^x) extends to a meromorphic function 
on C with possible poles located in |(1 — No). The order of a pole is at most 
2 sd(x). 
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6. Symmetry reduction 


Suppose that the representation {V,x) of Fa decomposes into a direct (finite) sum 

m 

(10) X = 0Xi 

i=i 

of representations {Vj,Xj) of F^. Then the Selberg zeta function factors accordingly 
[36, Theorem 7.2]: 

m 

( 11 ) Z{s,x) = Y[Z{s,Xj)- 

The structure of the interaction of the representation x with the action Ts yields 
that this kind of factorization already happens at the level of the transfer operators, 
and that (11) is an immediate consequence of this more general result. 


Theorem 6.1. Suppose that x decomposes as in (10), and let £s,x ® transfer 
operator of the form as in Section 4 or 5. Then 


( 12 ) 


- 

1=1 


Xj ■ 


Moreover, 


(13) 


Z{s,x) = Y[Z{s,Xj)- 

j=i 


Proof. Obviously, we have 

m m 

Fct{D;V) = 0 Fct(i:i;F,) and B{V;V) 

1=1 1=1 

For any 5 £ Fa and any function /, the map Ts{g) only acts on the argument of /, 
while xid) only acts on the vector f{z). Therefore, 

m m 

(s) = x{9)tY{ 9 ) = 0 Xj{g)T-Y^ { 9 ) = 0 aY {g)- 

1=1 1=1 

This implies (12). Then (13) follows immediately from 
Z{s,x) = det(l - £s.x) 

m m 

= det (1 — £s,xj) = det(l — Cs,xj) 

1=1 1=1 

m 

= Wz{s,Xj)- 
1=1 


□ 
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7. Billiard flows 


Recall the elements 


Q 


0 1 
1 0 


and 



0 

1 


in PGL 2 (M). The discrete dynamical systems and the transfer operators from 
Sections 4 and 5 commute with the action of Q and as{Q) for with A < 2 
respectively with the action of J and as{J) for with A > 2. As shown in [21] for 
Fa with A < 2 with the trivial representation (C,id) this exterior symmetry allows 
us to characterize even/odd Maass cusp forms as those period functions that are 
invariant/anti-invariant under Ts(Q), and it results in a factorization of the Selberg 
zeta function. 

An approach from a higher point of view was taken on in [28] and [31], where r> 
was extended by these symmetries and then the billiard flow was considered. In 
the following we briefly survey this approach and extend it to include non-trivial 
representations. 

Let 

fx:={Q,Tx) = {J,rx) <PGL 2 (M) 


denote the triangle group underlying the Hecke triangle group. Further, we extend 
the representation (P, y) of Pa to a representation of Pa, also denoted by X) by 
defining xiQ) equivalently, xi^))- 

We consider the billiard flow on rA\IHI, that is, the geodesic flow on rA\IHI considered 
as an orbifold. We inherit the cross sections for the geodesic flow on rA\IHI to rA\lHI 
by taking “half” of it and then use the arising discrete dynamical systems to define 
the transfer operator families. For example, the cross section for Fa, A > 2, is only 
C'^ from Section 4.2 respectively 4.3. For Fa with A = 2cos7r/g and q odd, it is 


(14) C" := {u e I Rebase(p) = 0, 7 j,(oo) G (0, Ijst, 7i,(-oo) G (-oo, 0)st}- 


For Fa with A = 2cos'!T/q and q even, the point 1 G R is an endpoint of periodic 
geodesics. For that reason one cannot simply use (14) as cross section but rather 
needs an average over all reasonable choices for C to achieve that the tangent 
vectors belonging to these special periodic geodesics are “inner vectors”. We refer 
to [28] for details. The upshot is that the contribution from the element gq /2 gets 
an additional weight of 1/2. We also refer to [28, 31] for the explicit formulas for the 
arising discrete dynamical systems which only need to be extended by the obvious 
weights. The discrete dynamical systems for F 2 are constructed in an analogous 
way. 

To keep this article to a reasonable length, we only state the results for some cofinite 
Hecke triangle groups. For the others, analogous results are true. Recall that for 
5 G Fa we have N{g) = N{g‘^y/‘^. 

Theorem 7.1. Let Tx be a cofinite Hecke triangle group with A = 2cos^, q odd. 
Set m := 
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(i) 


(ii) 


(iii) 


The transfer operator associated to the slow discrete dynamical system for Fa 

is 

q-l 

^tT= Y.^^(9k) + as{Qgk), 

k—m 

acting on Fct((0,1); V). 

Suppose that V = C and x o,cts trivially onVx- 

(a) If xiQ) = 1 Res € (0,1), then the even Maass cusp forms for Fa are 
isomorphic to the eigenfunctions with eigenvalue 1 of Cffff that satisfy 
the regularity conditions from Theorem 4-T 

(b) IfxiQ) = and Res S (0,1), then the odd Maass cusp forms for Tx are 
isomorphic to the eigenfunctions with eigenvalue 1 of Cfff" that satisfy 
the regularity conditions from Theorem ^-l- 

The transfer operator associated to the fast discrete dynamical system for Fa 

is 


rfast _ 


0is{Qgq-i) 0is{gk) + as{Qgk) 

n^N k—m 

E ots{g^_i) + as{Qg'f_i) E as{gk) + a^iQgk) 

n^N k—m > 


acting on B{T~^-Sq-i] V) ® B{T~^-Sr] V). For Res > they define nuclear 
operators of order 0, and s i—>■ extends meromorphically to all of C with 

poles located as in Theorem 5.1 of order at most 2sd(x)- 
(iv) For Re s > 1 we have 


OO 

det(l - £{“*) = n n det (l - Xig) det/iV( 5 )-(^+'=)) , 

[9]e[r,,]pfe=o 


which is a dynamical zeta function. 


Proof. These statements (and also the corresponding statements for the other Hecke 
triangle groups) are easily adapted from [28, 31]. □ 

In Theorem 7.1(ii) we see that this specific choice of representation mimicks Dirich- 
let ixiQ) = 1) respectively Neumann (x(Q) = — 1) boundary conditions. The 
same interpretation also holds for the fast transfer operators and their dynamical 
zeta functions, for which we refer to [28]. We remark that these results allow us 
to reformulate the Phillips-Sarnak conjecture on the non-existence of even Maass 
cusp forms in terms of non-existence of non-trivial highly regular 1-eigenfunctions 
of and 

It would be interesting to see if there is a similar interpretation for more general 
representations, in particular if dimR > 1. 
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